In the field of computing hardware used for processing data in an environment such as fault-tolerant learning machines, autonomous control, pattern matching, artificial intelligence, robotics, etc., much excitement has been generated by neural network models that have the capability to learn. One particular model, delta-backpropagation (DB), has been successfully taught to perform a wide variety of tasks. DB shows promise for rapidly performing tasks that traditionally require great computational resources (e.g., image processing, pattern completion, and searching), because its neural network algorithm consists of many simple processing elements all working in parallel rather than one central processing element (i.e. the computer) working in a serial fashion as occurs in other forms of computing.
To date, all DB studies have been accomplished in a simulated environment using serial computers, or computers with a limited number of parallel processors. Thus DB has not actually been used for such high powered applications in normal and everyday use as would be desirable. In such simulated environments, digital computers are programmed to simulate the neural network DB algorithm. Consequently, the speed advantage inherent in the DB model is lost and simulation studies may take days or even weeks to run. A fully parallel hardware implementation is necessary for determining the utility of DB in solving large-scale computationally-intensive problems. Furthermore, the DB elements must be implemented in VLSI technology in order to make feasible a system with great numbers of parallel elements.
A fully parallel hardware implementation may take the form of analog circuitry, digital circuitry, or a hybrid of the two. While a digital implementation may hold the advantage of higher precision with respect to mathematical computations, an analog system may be significantly simpler in terms of number of transistors and, consequently, more processing elements may fit onto a VLSI chip of a given area. The problem then is how to implement the delta backpropagation algorithm in an analog hardware form that lends itself to implementation in VLSI.
Essentially being modeled after the human brain, neural networks generally consist of a number of simple processing elements, called "neurons", that are connected by conductive elements called "synapses". The conductance of the synapses are continuously variable. Information is stored in these systems by synapse conductance values. One popular prior art scheme for connecting neurons and synapses is depicted in FIG. 1. The network 10 is prompted by applying analog or digital signals to the input lines 12. This activates the neurons 14 and synapses 16 in the network 10. The degree to which a given neuron 14 is activated depends on the activation of the neurons 14 in the previous layer as well as the conductances of the weights leading to that neuron 14. After the system has settled, the output nodes 18 give the result. An electrical realization of such a prior art feedforward system is depicted in FIG. 2. The synapses 16 can be implemented as resistors and the neurons 14 as summers and threshold functions connected in series. Note that this configuration consists of three layers. The input layer, which can be as simple as a buffer or even a direct connection, is required to excite the first layer of synaptic elements. One or more hidden layers are required if the network 10 is to be capable of solving certain classes of problems. The output layer is required to sum the information from the hidden layer units, and possibly to threshold the resulting signals. Note that the number of layers, as well as the number of neurons 14 in each layer, are variables which must be selected by the neural network designer according to the task that the network 10 is to carry out.
While such a feedforward system can be used once the synaptic weights (i.e. conductances) have been set, a major consideration of neural network design is how to adjust these weights. A popular method of weight adjustment is the delta-backpropagation method. In this method, the network is trained by example. For a particular task, the network is repeatedly trained by applying representative input values and simultaneously applying the associated desired target output values to the network. A backpropagation system is then used to modify the weights such that the target output is more likely to occur given the applied input. Because the weights cannot be changed greatly during each backpropagation pass (otherwise previously stored information may be corrupted), many thousands or even millions of backpropagation training passes may be necessary to fully train such a network.
One prior art attempt at solving the problem addressed by the present invention employed VLSI capacitive elements for storing weights as voltages. This approach, of course, has the great disadvantage that the capacitances tend to discharge with time. Thus, the circuit has to be kept at a low temperature so as to minimize charge leakage. Even despite such precautions, however, the charges will dissipate slowly such that the weights represented by the capacitive charges need to be regenerated every day or two. This is intolerable for most application and, therefore, a more permanent storage of weighting values is highly desirable, such as that provided by resistive elements in other neural network applications. In this regard, however, perhaps the most serious obstacle to designing a practical DB processor is the lack of a suitable programmable resistive memory (RPM). Such memory elements, of course, are necessary to connect the various processing nodes of a neural network and store the information employed in the network. Among the required characteristics of such a PRM are high resistance, fast programmability, and non-volatility. While such devices have yet to go into actual production, work is well underway at the Jet Propulsion Laboratory (JPL) in Pasadena, Calif. and other research facilities with respect to the production of practical PRMs. Prototype PRM elements have been fabricated in prototype form using thin-film deposition techniques; and, while these devices are not yet fast enough to be used in a DB system, the advancements made to date suggest that memory elements with the required characteristics on a commercial basis may not be too far off.